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Abstract 


A post-introduction evaluation (PIE) is the overall assessment of the impact of the introduction of a new vaccine(s) on a 
country’s national immunization programme. It rapidly identifies problem areas needing correction within the 
immunization programme either pre-existing or resulting from the introduction of a new vaccine, and provides valuable 
lessons for future vaccine introductions. The World Health Organization (WHO) has developed a comprehensive manual 
and a tool to guide country teams to conduct PIE effectively. 

I have been involved in four PIE activities and have come up with these suggestions, that to achieve the objectives of PIE 
there is the need for (i) post PIE follow-ups, (ii) review and simplification of the PIE tool and (iii) a look at the number of 
PIEs one country can conduct irrespective of how many times a new vaccine is introduced in the country. This script is 


not intended to undermine the authenticity and modalities of the PIE structure but to enhance the achievements of the 


purpose and objectives of it. 
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Background 


A post-introduction evaluation (PIE) according to the 
WHO manual [1] is the assessment of the overall impact 
of the introduction of a new vaccine(s) on a country’s 
national immunization programme. It focuses on a range 
of programmatic aspects, such as_ pre-introduction 
planning, vaccine storage and wastage, logistics of 
administering the vaccine, and community receptiveness 
to the vaccine. It rapidly identifies problem areas needing 
correction within the immunization programme either 
pre-existing or resulting from the introduction of a new 
vaccine, and provides valuable lessons for future vaccine 
introductions. WHO recommends that all countries which 
have introduced a new vaccine conduct a PIE. WHO has 
developed a comprehensive manual and to tool to guide 
country teams to conduct PIE effectively. The manual 
suggests that, a PIE should be done between 6 and 12 
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months after introduction of the new vaccine. The PIE 
methodology consists of using questionnaires, checklists, 
observation of practices and recording reviews, at all 
levels of the health service, including cold and dry vaccine 
storage areas and points of vaccine administration, such 
as health facilities. 


I have been involved in four PIE activities; two in Ghana 
[2] Eritrea [3] and Nigeria [4]. The process at each level - 
national, regional, district and the health centre took 2-3 
hours administering a questionnaire and observing 
practical activities. Key activities conducted were (i) 
administration of questionnaire as already said, (ii) 
physical inspection of the cold chain especially vaccine 
stock, temperature monitoring, records keeping, waste 
disposal sites (incinerators and pits); (iii) observation of 
practical immunization sessions and exit interviews with 
care givers (iv) preparation of comprehensive report with 
recommended actions to improve challenges and gaps in 
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the immunization programme of each country. This script 
is not intended to undermine the authenticity and 
modalities of the PIE structure but to enhance the 
achievements of the purpose and objectives of it. 


Introduction Pathway 


Ghana introduced measles second dose (MSD), 
pneumococcal conjugate vaccine (PCV) and rotavirus 
(Rota) vaccine in 2012 and conducted the PIE in 2013. 
Ghana will introduce Meningococcal A conjugate vaccine 
(Men A) in November 2016. Eritrea introduced MSD and 


Rota in 2014 and conducted PIE in February 2015 and 
planned to introduced PCV and inactivated Polio vaccine 
(IPV) in July same year (2015). Nigeria introduced IPV in 
2015 and started introduction of PCV-10 in some States as 
a start, and conducted PIE in March 2016. Nigeria will 
continue with the introduction of PCV in the remaining 
states. 


Tables 1-3 are some of the recommendations from the PIE 
conducted in Ghana (2013); Eritrea (2015) and Nigeria 
(2016). 


1 Training 


All documents related to new vaccines (MCV2, PCV, and 
rotavirus) should be made available at all levels 


Organize continuous refresher training for new and old 
staff on injection techniques, policy on open vials etc. 


2 Coverage, Recording and Reporting 


Ensure that sufficient quantities of updated reporting 
forms are available and used at outreach and static 
sessions 


New immunization wall monitoring charts should be 
provided to all areas that donot have them 


Updated tools such as tally sheets and health cards should 
be widely distributed 


3 Cold Chain e 


Repair/replace non-functional refrigerators quickly 


4 Monitoring/ Supervision 


Streng then the supervision component of the EPI 
programmes in order to maintain effective implementation 
of immunization activities. 


Supervisors should provide written feedback during visits; 
Standardize supervisory checklists to include EPI. 


5 Waste Management e 


Fence around all pits and incinerators. 


6 Vaccine management 


Vaccine wastage should be tracked and reported in monthly 
reports. 


Guidelines on vaccine management currently in the training 
manual should be put in a document of its own and 
disseminated. 
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Adverse Events Following 
Immunization (AEFI). 


Provide written guidelines to Districts and HFs; HCW 
should remember to sensitize caregivers and mothers 
about possible adverse events 


Encourage community and HF to report AEF 


Advocacy and Communication 


Provide educational materials in local languages. 


Source -Ghana PIE report - March 2014 


Table 1: Ghana (2013) - PIE of Rotavirus, PCV and MSD. 


Pre-implementation Planning and 
vaccine introduction process - 


All relevant documents need to be updated with Rotavirus 
and MCV2 and also with PCV and IPV and shared with all 
levels ahead of the new vaccines scheduled for introduction 
in July 2015 


National orientation on schedule for rotavirus among new 
staff is urgently required to harmonize knowledge on the 
schedule. 


Training 


Since PCV13 and IPV are planned for simultaneous 
introduction, care must be taken during the combined 
training in order not to confuse the health workers 


Vaccine coverage and vaccine 
wastage monitoring 


Orientation for field staff is important and this can be 
through supportive supervision to the regions and health 
facilities or once a year refresher training at all levels. 


Vaccine management, Transport 
and Logistics 


Consider the bundling system to prevent shortages of some 
essential items in future. 


Waste management and injection 
Safety 


Supportive supervision is required to put corrective 
measures in place 


Monitoring and Supervision 


WHO and Unicef to support national EPI team to develop 
supervisory checklist, and facilitate some of the trips on 
supervisory visits to the lower levels. 


AEFI 


Strengthen collaboration between ADR program me and 
EPI for Surveillance for AEFI for AEFI detection, report and 
management at the health facilities 


Advocacy, Communications, and 
Community acceptance 


There is need to increase the awareness of the health 
benefits of immunization among the general public 


Lessons for future introduction 


Copies of national introduction plans for PCV13 and IPV 
should be shared with regions. 


Source -Eritrea PIE report - April 2015 


Table 2: Eritrea (2015) PIE of Rota, and MSD. 
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1 Training 


States and Local Government Authorities (LGAs) in collaboration with local 
partners should conduct refresher trainings for immunization on the 
identified gaps. In case of cascade trainings, step down trainings should be 
followed up; 


States and LGAs teams should use monthly LGAs Immunization Officers and 
healthcare worker meetings as opportunity to train healthcare workers on 
data management, immunization formulae and archiving of routine 
immunizationdocuments; 


Performance 
Monitoring 


National Public Health and Child Development Authority (NPHCDA) should 
review and develop a monitoring chart that includes more than one antigen 
(in case for antigens with less than 3 doses). 


State, LGA and health facilities should develop PCV-10 and IPV monitoring 
charts including any other new vaccines introduced; 


Vaccine 
Management 


NPHCDA in collaboration with partners should procure sufficient fridge and 
freeze tags; 


States and LGAs should facilitate distribution of these temperature 
monitoring devices to all health facilities with vaccines refrigerators; 


4 Cold Chain 


States and LGAs with support from NPHCDA should carry out bi-annual 
cold chain assessment, establish the number of the broken down 
equipment and repair accordingly. The assessment should also provide 
data for replacement of aged/obsolete equipment. 


All LGAs should have cold and dry stores for the storage of vaccines and 
injection materials respectively; 


All States, LGAs and Health facilities using electricity as a source of power 
to run refrigerators, should have a backup source in case of power outage. 


Cold chain equipment (Walk-In-Cold-Rooms (WICRs), Refrigerators and 
freezers) using grid electricity as source of energy should be supplied and 
connected with surge regulators to avoid damages. 


6 AEFI 


States and LGAs should conduct clinical sensitizations on AEFI and ensure 
revitalization of AEFI committees. Prior to sensitization, States distribute 
AEFI booklets/protocols/kits to all LGAs and subsequently to all facilities. 


NPHCDA should consider to incorporate the AEFI reporting as a key 
indicator to be reported for performance based financing schemes 


7 Waste management 


NPHCDA should finalize review of waste management policy/policy 
guidelines and disseminate to the States and LGAs for implementation. 


All states should install incinerators and conduct regular on site supportive 
supervision by management to ensure incinerators are well operated and 
maintained. 
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Social mob/ 
Communication 


e States/LGAs should translate IEC materials into local languages, pretest and 
distribute to all health facilities. 


e Best practices for social mobilization from other LGAs for immunization 
awareness creation can be shared, such practices include, songs during 
health talks to raise awareness of caregivers for vaccination services. 


e NPCHDA at central level should prepare generic IEC Materials in English 
and distribute to states. 


Supportive 
supervision 


e The NPHCDA, States and LGAs EPI teams should ensure effective regular 
supervisory visits is conducted by Government Officials, action points are 
documented and followed up for their implementation. 


10 Data management 


e Data Quality Self-Assessments (DQSA) should be conducted to improve 
data quality during supportive supervision. 


e States should encourage LGAs to conduct routine immunization 
performance review meetings, prioritize health facilities by the 
performances and provide support accordingly. 


e = Health facilities reporting vaccination coverage less than 80% or above 
100% consistently, should implement REW strategy and DQSA. 


Source -Nigeria PIE report - March 2016 


Table 3: Nigeria (2016) - PIE of IPV and PCV. 


Discussion 


Whilst contemplating on how to make sense (effective 
and efficient) of PIE, the following issues came up for 
discussion and are worthy of consideration. 


Should there be PIE in a country for every new 
vaccine introduced -This may not be necessary because 
there would have been enough information from the last 
PIE to guide the introduction of the new vaccine. For 
example as these three countries have conducted PIE for 
any of these new vaccines: Rota, MSD PCV or IPV then 
there should be no need when the same countries 
introduce Men a vaccine using the same rigorous PIE 
process. A simplified questionnaire focusing simply on 
cold chain expansion, training, social mobilization, AEFI 
monitoring, recording tools is enough. Issues like 
coverage, wastage, dropout etc and other detailed 
information as before may not be relevant. 


Should PIE for same vaccine be conducted in every 
country-This may also not be necessary because 
previous report in one country can be used to guide the 
introduction of the vaccine in another country. There is 
the need to share PIE reports from neighbouring 
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countries planning to introduce same vaccines. For 
example if country A has conducted PIE following 
introduction of PCV and country B is also planning to 
introduce same vaccine (PCV), then the PIE report from 
country A should guide country B to minimize the 
weaknesses observed in country A. This also suggests that 
reports from Ghana, Eritrea and Nigeria and other PIEs 
should facilitate effective introduction of new vaccines in 
other countries.Rigorous PIE may not be required but a 
more flexible tool as suggested above may then be 
required insome of these countries following experiences 
that are available. 


Should the PIE tool be maintained as it is?-The current 
PIE tool uses close to three hours to conduct an 
assessment in one facility and this can be very stressful to 
both the facility team and the evaluators. The Tool, as it is 
now, is overloaded with repeated and overlapping 
questions. The specifics cannot be explained in this script 
but a technical committee to review and simplify it would 
be more useful to all parties. Experience from the PIEs so 
far conducted shows that some of the questions can be 
taken out. It appears PIE is gradually replacing 
comprehensive EPI Review. To modify PIE to cover issues 
required in traditional EPI Review is a good innovation if 
so then the arguments above hold that PIE in its current 
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form should not be conducted repeatedly and frequently 
in one country. 


What happens after PIE - Some ideas for post PIE are 
here for discussion. (i) Countries need to develop follow 
up improvement plans to address the gaps identified by 
the PIE. (ii) HQ/AFRO /IST must share PIE reports with 
countries especially reports that will enhance effective 
introduction of particular vaccines as indicated above. 
(iii) HQ/AFRO/IST must have post PIE assessment plan to 
follow up on countries improvement plans. This should be 
a very simple checklist taking not more than an hour to 
conduct at the facility to encourage countries to make 
sure most of the recommendations from the PIE are 
addressed. 


Conclusion 


PIE is a relevant component of the new vaccine 
introduction process and must be made to remain 
relevant and useful to countries. It reveals the strengths 
and weaknesses of the immunization programme in the 
country. Its objectives are to: 


a. Document the process of introduction of new 
vaccine(s) into the immunization program. 


b. Identify operational, logistical and environmental 
challenges associated with the introduction of the 
Vaccine (s) and provide recommendations to address 
them 


c. Inform future program planning and implementation 
in the same setting, or in other 
Geographic areas/ countries from the lessons drawn 
from the PIE. 
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To achieve these objectives, there is need for: 


a. 


b. 


post PIE follow-ups, 
review and simplification of the current PIE tool and 


A look at the number of PIEs one country can conduct 
irrespective of how many times a new vaccine is 
introduced in the country. These will enhance the 
achievements of the purpose and objectives of PIE. 
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